27 research outputs found
Anomaly detection with semi-supervised classification based on risk estimators
A significant limitation of one-class classification anomaly detection
methods is their reliance on the assumption that unlabeled training data only
contains normal instances. To overcome this impractical assumption, we propose
two novel classification-based anomaly detection methods. Firstly, we introduce
a semi-supervised shallow anomaly detection method based on an unbiased risk
estimator. Secondly, we present a semi-supervised deep anomaly detection method
utilizing a nonnegative (biased) risk estimator. We establish estimation error
bounds and excess risk bounds for both risk minimizers. Additionally, we
propose techniques to select appropriate regularization parameters that ensure
the nonnegativity of the empirical risk in the shallow model under specific
loss functions. Our extensive experiments provide strong evidence of the
effectiveness of the risk-based anomaly detection methods
Scoring Functions for Multivariate Distributions and Level Sets
Interest in predicting multivariate probability distributions is growing due
to the increasing availability of rich datasets and computational developments.
Scoring functions enable the comparison of forecast accuracy, and can
potentially be used for estimation. A scoring function for multivariate
distributions that has gained some popularity is the energy score. This is a
generalization of the continuous ranked probability score (CRPS), which is
widely used for univariate distributions. A little-known, alternative
generalization is the multivariate CRPS (MCRPS). We propose a theoretical
framework for scoring functions for multivariate distributions, which
encompasses the energy score and MCRPS, as well as the quadratic score, which
has also received little attention. We demonstrate how this framework can be
used to generate new scores. For univariate distributions, it is
well-established that the CRPS can be expressed as the integral over a quantile
score. We show that, in a similar way, scoring functions for multivariate
distributions can be "disintegrated" to obtain scoring functions for level
sets. Using this, we present scoring functions for different types of level
set, including those for densities and cumulative distributions. To compute the
scoring functions, we propose a simple numerical algorithm. We illustrate our
proposals using simulated and stock returns data
A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition
Multi-step ahead forecasting is still an open challenge in time series
forecasting. Several approaches that deal with this complex problem have been
proposed in the literature but an extensive comparison on a large number of
tasks is still missing. This paper aims to fill this gap by reviewing existing
strategies for multi-step ahead forecasting and comparing them in theoretical
and practical terms. To attain such an objective, we performed a large scale
comparison of these different strategies using a large experimental benchmark
(namely the 111 series from the NN5 forecasting competition). In addition, we
considered the effects of deseasonalization, input variable selection, and
forecast combination on these strategies and on multi-step ahead forecasting at
large. The following three findings appear to be consistently supported by the
experimental results: Multiple-Output strategies are the best performing
approaches, deseasonalization leads to uniformly improved forecast accuracy,
and input selection is more effective when performed in conjunction with
deseasonalization
Machine learning strategies for multi-step-ahead time series forecasting
How much electricity is going to be consumed for the next 24 hours? What will be the temperature for the next three days? What will be the number of sales of a certain product for the next few months? Answering these questions often requires forecasting several future observations from a given sequence of historical observations, called a time series. Historically, time series forecasting has been mainly studied in econometrics and statistics. In the last two decades, machine learning, a field that is concerned with the development of algorithms that can automatically learn from data, has become one of the most active areas of predictive modeling research. This success is largely due to the superior performance of machine learning prediction algorithms in many different applications as diverse as natural language processing, speech recognition and spam detection. However, there has been very little research at the intersection of time series forecasting and machine learning.The goal of this dissertation is to narrow this gap by addressing the problem of multi-step-ahead time series forecasting from the perspective of machine learning. To that end, we propose a series of forecasting strategies based on machine learning algorithms.Multi-step-ahead forecasts can be produced recursively by iterating a one-step-ahead model, or directly using a specific model for each horizon. As a first contribution, we conduct an in-depth study to compare recursive and direct forecasts generated with different learning algorithms for different data generating processes. More precisely, we decompose the multi-step mean squared forecast errors into the bias and variance components, and analyze their behavior over the forecast horizon for different time series lengths. The results and observations made in this study then guide us for the development of new forecasting strategies.In particular, we find that choosing between recursive and direct forecasts is not an easy task since it involves a trade-off between bias and estimation variance that depends on many interacting factors, including the learning model, the underlying data generating process, the time series length and the forecast horizon. As a second contribution, we develop multi-stage forecasting strategies that do not treat the recursive and direct strategies as competitors, but seek to combine their best properties. More precisely, the multi-stage strategies generate recursive linear forecasts, and then adjust these forecasts by modeling the multi-step forecast residuals with direct nonlinear models at each horizon, called rectification models. We propose a first multi-stage strategy, that we called the rectify strategy, which estimates the rectification models using the nearest neighbors model. However, because recursive linear forecasts often need small adjustments with real-world time series, we also consider a second multi-stage strategy, called the boost strategy, that estimates the rectification models using gradient boosting algorithms that use so-called weak learners.Generating multi-step forecasts using a different model at each horizon provides a large modeling flexibility. However, selecting these models independently can lead to irregularities in the forecasts that can contribute to increase the forecast variance. The problem is exacerbated with nonlinear machine learning models estimated from short time series. To address this issue, and as a third contribution, we introduce and analyze multi-horizon forecasting strategies that exploit the information contained in other horizons when learning the model for each horizon. In particular, to select the lag order and the hyperparameters of each model, multi-horizon strategies minimize forecast errors over multiple horizons rather than just the horizon of interest.We compare all the proposed strategies with both the recursive and direct strategies. We first apply a bias and variance study, then we evaluate the different strategies using real-world time series from two past forecasting competitions. For the rectify strategy, in addition to avoiding the choice between recursive and direct forecasts, the results demonstrate that it has better, or at least has close performance to, the best of the recursive and direct forecasts in different settings. For the multi-horizon strategies, the results emphasize the decrease in variance compared to single-horizon strategies, especially with linear or weakly nonlinear data generating processes. Overall, we found that the accuracy of multi-step-ahead forecasts based on machine learning algorithms can be significantly improved if an appropriate forecasting strategy is used to select the model parameters and to generate the forecasts.Lastly, as a fourth contribution, we have participated in the Load Forecasting track of the Global Energy Forecasting Competition 2012. The competition involved a hierarchical load forecasting problem where we were required to backcast and forecast hourly loads for a US utility with twenty geographical zones. Our team, TinTin, ranked fifth out of 105 participating teams, and we have been awarded an IEEE Power & Energy Society award.Doctorat en sciences, Spécialisation Informatiqueinfo:eu-repo/semantics/nonPublishe
Machine learning strategies for multi-step-ahead time series forecasting
How much electricity is going to be consumed for the next 24 hours? What will be the temperature for the next three days? What will be the number of sales of a certain product for the next few months? Answering these questions often requires forecasting several future observations from a given sequence of historical observations, called a time series. Historically, time series forecasting has been mainly studied in econometrics and statistics. In the last two decades, machine learning, a field that is concerned with the development of algorithms that can automatically learn from data, has become one of the most active areas of predictive modeling research. This success is largely due to the superior performance of machine learning prediction algorithms in many different applications as diverse as natural language processing, speech recognition and spam detection. However, there has been very little research at the intersection of time series forecasting and machine learning.The goal of this dissertation is to narrow this gap by addressing the problem of multi-step-ahead time series forecasting from the perspective of machine learning. To that end, we propose a series of forecasting strategies based on machine learning algorithms.Multi-step-ahead forecasts can be produced recursively by iterating a one-step-ahead model, or directly using a specific model for each horizon. As a first contribution, we conduct an in-depth study to compare recursive and direct forecasts generated with different learning algorithms for different data generating processes. More precisely, we decompose the multi-step mean squared forecast errors into the bias and variance components, and analyze their behavior over the forecast horizon for different time series lengths. The results and observations made in this study then guide us for the development of new forecasting strategies.In particular, we find that choosing between recursive and direct forecasts is not an easy task since it involves a trade-off between bias and estimation variance that depends on many interacting factors, including the learning model, the underlying data generating process, the time series length and the forecast horizon. As a second contribution, we develop multi-stage forecasting strategies that do not treat the recursive and direct strategies as competitors, but seek to combine their best properties. More precisely, the multi-stage strategies generate recursive linear forecasts, and then adjust these forecasts by modeling the multi-step forecast residuals with direct nonlinear models at each horizon, called rectification models. We propose a first multi-stage strategy, that we called the rectify strategy, which estimates the rectification models using the nearest neighbors model. However, because recursive linear forecasts often need small adjustments with real-world time series, we also consider a second multi-stage strategy, called the boost strategy, that estimates the rectification models using gradient boosting algorithms that use so-called weak learners.Generating multi-step forecasts using a different model at each horizon provides a large modeling flexibility. However, selecting these models independently can lead to irregularities in the forecasts that can contribute to increase the forecast variance. The problem is exacerbated with nonlinear machine learning models estimated from short time series. To address this issue, and as a third contribution, we introduce and analyze multi-horizon forecasting strategies that exploit the information contained in other horizons when learning the model for each horizon. In particular, to select the lag order and the hyperparameters of each model, multi-horizon strategies minimize forecast errors over multiple horizons rather than just the horizon of interest.We compare all the proposed strategies with both the recursive and direct strategies. We first apply a bias and variance study, then we evaluate the different strategies using real-world time series from two past forecasting competitions. For the rectify strategy, in addition to avoiding the choice between recursive and direct forecasts, the results demonstrate that it has better, or at least has close performance to, the best of the recursive and direct forecasts in different settings. For the multi-horizon strategies, the results emphasize the decrease in variance compared to single-horizon strategies, especially with linear or weakly nonlinear data generating processes. Overall, we found that the accuracy of multi-step-ahead forecasts based on machine learning algorithms can be significantly improved if an appropriate forecasting strategy is used to select the model parameters and to generate the forecasts.Lastly, as a fourth contribution, we have participated in the Load Forecasting track of the Global Energy Forecasting Competition 2012. The competition involved a hierarchical load forecasting problem where we were required to backcast and forecast hourly loads for a US utility with twenty geographical zones. Our team, TinTin, ranked fifth out of 105 participating teams, and we have been awarded an IEEE Power & Energy Society award.Doctorat en sciences, Spécialisation Informatiqueinfo:eu-repo/semantics/nonPublishe
Conditionally dependent strategies for multiple-step-ahead prediction in local learning
Computational intelligence approaches to multiple-step-ahead forecasting rely on either iterated one-step-ahead predictors or direct predictors. In both cases the predictions are obtained by means of multi-input single-output modeling techniques. This paper discusses the limitations of single-output approaches when the predictor is expected to return a long series of future values, and presents a multi-output approach to long term prediction. The motivation for this work is that, when predicting multiple steps ahead, the forecasted sequence should preserve the stochastic properties of the training series. However, this may not be the case, for instance in direct approaches where predictions for different horizons are produced independently. We discuss here a multi-output extension of conventional local modeling approaches, and present and compare three distinct criteria for performing conditionally dependent model selection. In order to assess the effectiveness of the different selection strategies, we carry out an extensive experimental session based on the 111 series in the NN5 competition. © 2010 International Institute of Forecasters.SCOPUS: ar.jinfo:eu-repo/semantics/publishe
Recursive multi-step time series forecasting by perturbing data
info:eu-repo/semantics/publishe
On the Predictive Accuracy of Neural Temporal Point Process Models for Continuous-time Event Data
Temporal Point Processes (TPPs) serve as the standard mathematical framework
for modeling asynchronous event sequences in continuous time. However,
classical TPP models are often constrained by strong assumptions, limiting
their ability to capture complex real-world event dynamics. To overcome this
limitation, researchers have proposed Neural TPPs, which leverage neural
network parametrizations to offer more flexible and efficient modeling. While
recent studies demonstrate the effectiveness of Neural TPPs, they often lack a
unified setup, relying on different baselines, datasets, and experimental
configurations. This makes it challenging to identify the key factors driving
improvements in predictive accuracy, hindering research progress. To bridge
this gap, we present a comprehensive large-scale experimental study that
systematically evaluates the predictive accuracy of state-of-the-art neural TPP
models. Our study encompasses multiple real-world and synthetic event sequence
datasets, following a carefully designed unified setup. We thoroughly
investigate the influence of major architectural components such as event
encoding, history encoder, and decoder parametrization on both time and mark
prediction tasks. Additionally, we delve into the less explored area of
probabilistic calibration for neural TPP models. By analyzing our results, we
draw insightful conclusions regarding the significance of history size and the
impact of architectural components on predictive accuracy. Furthermore, we shed
light on the miscalibration of mark distributions in neural TPP models. Our
study aims to provide valuable insights into the performance and
characteristics of neural TPP models, contributing to a better understanding of
their strengths and limitations
A gradient boosting approach to the Kaggle load forecasting competition
We describe and analyse the approach used by Team TinTin (Souhaib Ben Taieb and Rob J Hyndman) in the Load Forecasting track of the Kaggle Global Energy Forecasting Competition 2012. The competition involved a hierarchical load forecasting problem for a US utility with 20 geographical zones. The data available consisted of the hourly loads for the 20 zones and hourly temperatures from 11 weather stations, for four and a half years. For each zone, the hourly electricity loads for nine different weeks needed to be predicted without having the locations of either the zones or stations. We used separate models for each hourly period, with component-wise gradient boosting for estimating each model using univariate penalised regression splines as base learners. The models allow for the electricity demand changing with the time-of-year, day-of-week, time-of-day, and on public holidays, with the main predictors being current and past temperatures, and past demand. Team TinTin ranked fifth out of 105 participating teams. © 2013 International Institute of Forecasters.SCOPUS: ar.jinfo:eu-repo/semantics/publishe